Differences in Time Delay between Search Engine Crawlers at Web Sites

Web log mining provides tremendous information about user traffic and search engine behavior at web sites. The behavior of search engines could be used in analyzing server load, quality of search engines, dynamics of search engine crawlers, ethics of search engines etc. Search engine crawlers are highly automated programs which are seldom regulated manually. These crawlers periodically visit the web sites to collect the information. The dynamicity of search engine crawlers could be identified with the time delay between two consecutive visits. The more the visits of a crawler to a web site, the more it contributes to the server load. We intend to see whether there is a significant difference in the time delay between visits of a search engine crawler. Similarly the time delay between visits of various search engine crawlers is also analyzed to identify the differences in their behavior.